Image processing method and apparatus for correlating a test image with a template

ABSTRACT

The image correlation method and apparatus correlates or matches a test image with a template. The image correlation apparatus includes an image processor for partitioning the template into a number of labels, for determining the total number of pixels N T  which form the template and for determining the number of pixels N i  which form each of the labels i. The image correlation apparatus also includes comparison means for comparing the test image to the template. The comparison means determines, for each predetermined gray level j, the number of pixels of the test image N j ,i representative of a predetermined gray level j which correspond to a predetermined label i of the template. The comparison means also determines, for each predetermined gray level j, the number of pixels of the test image N j  representative of a predetermined gray level j which correspond to the template. The image correlation apparatus further includes correlation means for determining the correlation X between the test image and the template according to a predetermined equation which is based, at least in part, upon N j ,i, N j , N i  and N T . The image correlation apparatus can also include an address generator for creating a number of relative offsets between the template and the test image. Thus, the test image can be compared to the template at each relative offset and the relative offset which provides the greatest correlation therebetween can be determined. Consequently, the test image and the template can be effectively matched such that a preselected object designated within the template can be located and identified within the test image.

This is a divisional of U.S. patent application Ser. No. 08/583,598, nowU.S. Pat. No. 5,809,171.

FIELD OF THE INVENTION

The present invention relates generally to image processing methods andapparatus and, more particularly, to image processing methods andapparatus which correlate a test image with a template.

BACKGROUND OF THE INVENTION

It is desirable in many instances to match or otherwise correlate a testimage with a template. For example, the test image may be an image whichwas recently obtained or captured, while the template may be based upona reference image obtained at some prior time. By matching the testimage to the template, specific objects or targets which are designatedin the template can be identified within the test image. This ability toidentify specific objects within the test image which have previouslybeen designated within a template is quite useful in a number ofapplications, including robotic control and other machine visionapplications, automated inspection systems, automated surveillancesystems and medical imagery applications. More specifically, in missileguidance applications, it is particularly desirable to automaticallyidentify a target within a test image which has previously beendesignated in a template or a reconnaissance photograph.

A number of image correlation and image detection methods have beendeveloped to identify previously designated objects within a test image.For example, a number of image correlation systems have been developedwhich detect edges or other boundaries within the test image. As knownto those skilled in the art, edges or boundaries of objects tend tocreate intensity discontinuities within the test image. Accordingly, byidentifying the edges or boundaries of objects, an outline of theobjects can sometimes be created. Thereafter, the edge detection systemsgenerally attempt to recognize previously designated objects based upontheir spatial relationship to an edge or other boundary which has beenidentified within the test image.

Another common image detection method is the Hough method. The Houghmethod is particularly applicable if little, if any, information isknown regarding the relative location of an edge or other boundarywithin the test image, but if the shape of the designated object can bedescribed as a parametric curve since the Hough method is well suited todetect such curves. The Hough method, as well as other image detectionmethods, are described in a book entitled Computer Vision by Ballard, etal., published by Prentice-Hall, Inc., Englewood Cliffs, N.J. (1982).

Although extensive research has been conducted to efficiently implementimage detection systems, such as edge detection systems and systemsbased upon the Hough method as described above and in more detail in thebook entitled Computer Vision, these conventional image detection andcorrelation systems still suffer from several deficiencies. For example,many conventional image detection methods require significant amounts ofdata processing and computation in order to effectively compare the testimage to the template or reference image. Thus, image detection systemsbased upon these conventional detection methods may require asignificant amount of time in order to compare the test image to thetemplate and to identify the designated object within the test image.While the delays engendered by these lengthy comparison processes areacceptable in many applications, some applications, such as roboticcontrol or other machine vision applications and missile guidanceapplications, typically require a relatively rapid comparison of thetest image to the template and, in some instances, may demand a nearreal time comparison of the test image to the template in order toeffectively identify the designated object or target.

In addition, at least some of these conventional image detection systemsrequire the test image as well as the reference image from which thetemplate is created to be obtained under relatively similar lighting andother environmental conditions in order to properly compare the testimage to the template. These environmental conditions include, amongother things, the viewpoint, i.e., the angle and direction, from whichthe template and the test image were obtained. If the lighting or otherenvironmental conditions change between the time at which the referenceimage from which the template is created is obtained and the time atwhich the test image is obtained, at least some of these conventionalimage detection methods require that the image depicted by the templatebe modeled.

For example, in missile guidance applications in which the template andthe test image are images of the terrain or landscape at a predeterminedlocation, conventional image detection methods generally require theviewpoint from which the reference image and the test image are obtainedto closely correspond in order to properly identify a designated objectwithin the test image. If, however, the lighting or environmentalconditions, such as the respective viewpoints, change between the timeat which the reference image is obtained and the time at which the testimage is obtained, the terrain at the desired location must generally bemodeled from the reference image according to a manual mensurationprocession. As known to those skilled in the art, the terrain at thedesired location can be mensurated by constructing three-dimensionalmodels of the landscape, including three-dimensional models of anybuildings of interest, at the desired location. As also known to thoseskilled in the art, the construction of appropriate three-dimensionalmodels generally requires the craftsmanship of an expert who is skilledand experienced in the mensuration process. Even with the assistance ofa mensuration expert, however, the construction of the appropriatethree-dimensional structures can consume hours or sometimes even days.

Once the terrain has been modeled based upon the reference image, thetemplate can be created based upon the model in order to compensate forthe variations in the lighting and other environmental conditionsbetween the time at which the reference image was obtained and the timeat which the test image is obtained. In this fashion, these conventionalimage detection systems can compare the template to the test image.

Conventional image detection systems also typically require thereference image and the test image to be obtained by sources of the samespectral type. For example, conventional image detection systems mayrequire both the reference and test images to be obtained by visiblewavelength sensors, such as photographic systems. Alternatively,conventional image detection systems may require both the reference andtest images to be obtained by infrared sensors or by synthetic apertureradar. In any instance, conventional image detection systems generallydo not permit the reference and test images to be obtained by differenttypes or cross-spectral sources. In addition, conventional imagedetection systems may restrict the types, shapes, locations or materialsof the objects which can be reliably designated and identified due, atleast in part, to limitations imposed upon the types of sensors whichcan be employed to create the reference and test images.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide animproved method and apparatus for correlating a test image with atemplate.

It is another object of the present invention to provide an improvedmethod and apparatus for identifying an object in the test image whichhas been designated within the template.

It is yet another object of the present invention to provide an imagecorrelation method and apparatus which rapidly compares a test imagewith a template.

It is still another object of the present invention to provide an imagecorrelation method and apparatus which reliably correlates a test imagewith a template which have been obtained under different lighting orother environmental conditions.

It is a further object of the present invention to provide an imagecorrelation method and apparatus which reliably correlates a test imagewith a template which have been obtained by different types orcross-spectral sources.

It is yet a further object of the present invention to provide an imagecorrelation method and apparatus which reliably identifies a designatedobject within a test image regardless of the type, shape or location ofthe designated object and regardless of the material from which thedesignated object is constructed.

These and other objects are provided, according to the presentinvention, by a method and apparatus for correlating a test image with atemplate which includes correlation means for determining thecorrelation X according to: ##EQU1##

Both the test image and template are formed of a number of pixelsrepresentative of respective predetermined gray levels. Thus, the imagecorrelation method and apparatus also includes template processing meansfor processing the pixels which form the template. The templateprocessing means includes means for partitioning the template into anumber of labels, each of which are formed of at least one pixel. Thetemplate processing means also includes means for determining the totalnumber of pixels N_(T) which form the template, and means fordetermining the number of pixels N_(i) which form each of the labels i.

The image correlation method and apparatus of the present invention alsoincludes comparison means for comparing the test image to the template,such as by overlaying the template on the test image, such that at leastsome of the pixels of the template correspond to respective pixels ofthe test image. The comparison means includes means for determining, foreach predetermined gray level j, the number of pixels of the test imageN_(j),i representative of a predetermined gray level j which correspondto a predetermined label i of the template. The comparison means alsoincludes means for determining, for each predetermined gray level j, thenumber of pixels of the test image N_(j) representative of apredetermined gray level j which correspond to the template. Based uponthe values of N_(T), N_(i), N_(j),i and N_(j), the correlation X betweenthe test image and the template can be determined according to: ##EQU2##wherein i_(min) and i_(max) are the minimum and maximum labeldesignations, respectively, and wherein j_(min) and j_(max) are theminimum and maximum gray levels, respectively. In order to efficientlycompute the correlation X, the image correlation method and apparatuscan also include means for determining ln N_(j),i and ln N_(j) prior todetermining the correlation of the test image with the template.

Based upon the above equation and the pixel-to-pixel comparison of thetest image and the template, the image correlation method and apparatusof the present invention can reliably correlate test images andtemplates which were obtained under different lighting or environmentalconditions, such as from different viewpoints. The image correlationmethod and apparatus of the present invention can also reliablycorrelate test images and templates which were obtained by differenttypes or cross-spectral sources and can readily identify designatedobjects within the test image regardless of the type, shape or locationof the designated object or the material from which the designatedobject is constructed.

According to one embodiment, the image correlation method and apparatusalso includes offset means for creating a number of relative offsetsbetween the template and the test image. According to this embodiment,the comparison means preferably compares the test image to the templateat each relative offset. Consequently, the comparison means preferablydetermines N_(j),i and N_(j) at each relative offset such that therespective correlation X between the test image and the template canalso be determined at each relative offset. Further, the imagecorrelation method and apparatus of this embodiment can also includemeans for determining the relative offset between the test image and thetemplate which provides the greatest correlation therebetween.Consequently, the test image and the template can be effectively matchedsuch that a preselected object designated within the template can belocated and identified within the test image.

According to one advantageous embodiment, the pixels which form the testimage and the template are assigned respective addresses. Thus, theimage correlation method and apparatus can include an address generatorwhich includes the offset means for creating the relative offset betweenthe template and the test image based upon the respective addresses ofthe pixels of the test image and the template.

The image correlation method and apparatus can also include rotationmeans for creating at least one relative rotational offset between thetemplate and the test image. According to this embodiment, thecomparison means compares the test image to the template at eachrelative rotational offset. More particularly, the comparison meansdetermines N_(j),i and N_(j) at each relative rotational offset suchthat the correlation X between the test image and the template at eachrelative rotational offset can also be determined. In addition, theimage correlation method and apparatus of this embodiment can includemeans for determining the relative rotational offset between the testimage and the template which provides the greatest correlationtherebetween.

In a similar fashion, the image correlation method and apparatus of thepresent invention can include scaling means for scaling the templaterelative to the test image according to at least one predeterminedscale. According to this embodiment, the comparison means compares thetest image to the template at each predetermined scale. Moreparticularly, the comparison means determines N_(j),i and N_(j) at eachpredetermined scale such that the correlation X between the test imageand the template can also be determined at each predetermined scale. Inaddition, the image correlation method and apparatus of this embodimentcan include means for determining the predetermined scale which providesthe greatest correlation between the template and the test image.

According to a further embodiment, the partitioning means of the imagecorrelation method and apparatus can include means for dividing thepredetermined gray levels into a plurality of bins, each of whichincludes a range of gray levels. The partitioning means of thisembodiment also includes means for determining, for each of the bins, anactual count of the number of pixels which are representative of graylevels within the range of gray levels included within the respectivebin. The partitioning means can also include means for assigning apredetermined minimum count to each bin which has an actual count whichis less than the predetermined minimum count. Further, the partitioningmeans can include means for reassigning the predetermined gray levels tothe plurality of bins such that the greater of the predetermined minimumcount or the actual count for each respective bin is within apredetermined range and, according to one advantageous embodiment, isequal. Following this reassignment, the pixels which are representativeof gray levels within each bin form a respective label.

According to one more specific embodiment, the partitioning means canalso include means for determining the maximum number of pixels of thetemplate which are representative of a single predetermined gray level.Therefore, the partitioning means can also include means for computingthe predetermined minimum count to be equal to a predeterminedpercentage, such as between about 15% and about 25%, of the maximumnumber of pixels which are representative of a single predetermined graylevel.

The image correlation method and apparatus of one particularlyadvantageous embodiment of the present invention can also includetemplate memory means and image memory means for storing indiciarepresentative of the pixels which form the template and the test image,respectively. In addition, the comparison means can include a number ofprocessing elements, such as field programmable gate arrays, whichcompare the template to the test image. In particular, the processingelements are adapted to compare, preferably concurrently, the test imageto the template at one or more of the relative offsets created by theoffset means. Thus, the image correlation method and apparatus of thepresent invention can efficiently and rapidly determine the relativeoffset between the template and the test image which provides thegreatest correlation therebetween.

The image correlation method and apparatus of the present invention canform a portion of a method and apparatus for recognizing or identifyinga preselected object within a test image. The object recognition methodand apparatus can also include template generation means for creating atemplate formed of a number of pixels which are thereafter partitionedinto labels, and an image capturing means for capturing the test image.The object recognition method and apparatus can also include objectdesignation means for designating a selected object within the templateand object recognizing means for recognizing or identifying thedesignated object within the test image. Once the image correlationmethod and apparatus has determined the relative positions of the testimage and the template which provide the greatest correlation Xtherebetween, the object recognition means selects the object within thetest image which corresponds to the designated object in the template.

In many instances, the template and the test image are obtained fromfirst and second viewpoints, respectively. Consequently, the imagerecognition method and apparatus can include an image processor forgeometrically warping and/or texture mapping the template to compensatefor differences between the first and second viewpoints prior to storingindicia representative of the plurality of pixels which form thetemplate in the template memory means. Further, the image processor canremove at least some of the background shading from the template priorto storing indicia representative of the pixels which form the templatein the template memory means in order to further improve the correlationprocess.

The image processor can also divide the template into a number of piecesin order to further compensate for deformation between the template andthe test image. According to this embodiment, the comparison meanscompares each piece of the template to a predetermined range of the testimage. The predetermined ranges of the template are selected such thatthe offset means of this embodiment can create respective relativeoffsets for each piece of the template which permit the adjacent piecesof the template to partially overlap or to be spaced apart by apredetermined gap while remaining within the respective predeterminedranges.

According to this embodiment of the present invention, the N_(j),idetermining means includes means for separately determining, for eachpiece of the template, the number of pixels of the test image N_(j),irepresentative of a predetermined gray level j which correspond to apredetermined label i of the respective piece of the template. Likewise,the N_(j) determining means of this embodiment includes means forseparately determining, for each piece of the template, the number ofpixels of the test image N_(j) representative of a predetermined graylevel j which correspond to the respective piece of the template. Inaddition, the correlation means of this embodiment separately determinesthe correlation X between the test image and each piece of the templateat each relative offset between the test image and the respective pieceof the template. Accordingly, the relative offset between the test imageand each respective piece of the template which provides the greatestcorrelation therebetween can be determined for each piece of thetemplate. Further, the image correlation method and apparatus of thisembodiment can include means for determining an overall correlationbetween the test image and the template by summing the respectivegreatest correlation determined for each piece of the template.

According to another embodiment of the object recognition method andapparatus of the present invention, the image capturing step meansincludes means for capturing a number of temporally distinct testimages. According to this embodiment, the comparison means includesmeans for separately comparing each of the temporally distinct testimages to the template such that the respective correlation X betweeneach of the temporally distinct test images and the template can beseparately determined. In addition, the means for determining therelative offset which provides the greatest correlation X between thetest image and the template can include means for determining, for eachrelative offset, a sum of the correlations X determined for each of thetemporally distinct test images at a respective offset which have beenweighted according to a predetermined time-based formula. In addition,the relative offset determining means can also include means fordetermining the relative offset between the test image and the templatewhich provides the greatest sum of weighted correlations X over time. Inthis fashion, the probability that the designated object is properlyrecognized within the test image is enhanced due to the separatecorrelation and comparison of a number of temporally distinct testimages with the template.

Therefore, the image correlation method and apparatus of the presentinvention can rapidly compare a test image to a template to determinethe correlation X therebetween with little, if any, modeling ormensuration required. In addition, the image correlation method andapparatus of the present invention can reliably correlate test imagesand templates which were obtained under different lighting or otherenvironmental conditions or which were obtained from cross-spectralsources. Further, the image correlation method and apparatus of thepresent invention can reliably correlate a test image with a templateregardless of the type, shape, location or material of the objectdesignated within the template. Consequently, the image correlationmethod and apparatus of the present invention provides a versatile,reliable and rapid comparison of a test image with a template such thatobjects which were previously designated within the template can bequickly and accurately identified in the test image.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of the presentinvention will be made apparent from the following detailed descriptionand from the drawings in which:

FIG. 1 is a schematic block diagram of the object recognition apparatus,including the image correlation apparatus, of one embodiment of thepresent invention.

FIG. 2 is a flow chart illustrating operations performed by the imagecorrelation method and apparatus of one embodiment of the presentinvention.

FIG. 3 is an exemplary histogram illustrating the number of pixels of atemplate which are representative of each of the predetermined graylevels.

FIG. 4 illustrates the exemplary histogram of FIG. 3 which has beendivided into a number of bins during the process of partitioning thetemplate into a number of labels.

FIG. 5 is a flow chart illustrating the operations performed topartition the template into a plurality of labels.

FIG. 6 is a simplified schematic depiction of a template which has beenpartitioned into three labels.

FIG. 7 illustrates the comparison of the template of FIG. 6 which hasbeen partitioned into three labels with a test image.

FIG. 8 illustrates the comparison of the template of FIG. 6 which hasbeen partitioned into three labels to a test image in which a relativerotational offset has been created between the test image and thetemplate.

FIG. 9 illustrates the comparison of a template of FIG. 6 partitionedinto three labels to a test image in which the template has beenenlarged by a predetermined scale.

FIG. 10 is a schematic block diagram of the image correlation apparatusincluding a plurality of parallel processing elements of one embodimentof the present invention.

FIG. 11 is a block diagram illustrating the operations performed by anobject recognition method and apparatus of one embodiment of the presentinvention which includes an image correlation method and apparatus.

FIG. 12 illustrates the comparison of a template to a test image inwhich the template has been divided into a number of pieces forcomparison with a predetermined ranges of the test image.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will now be described more fully hereinafter withreference to the accompanying drawings, in which a preferred embodimentof the invention is shown. This invention may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein; rather, this embodiment is provided sothat this disclosure will be thorough and complete and will fully conveythe scope of the invention to those skilled in the art. Like numbersrefer to like elements throughout.

The present invention provides, among other things, an image correlationmethod and apparatus for reliably correlating a test image with atemplate. Although the image correlation method and apparatus of thepresent invention is primarily discussed in conjunction with a targetrecognition system, such as for surveillance or missile guidanceapplications, the image correlation method and apparatus can be employedin a variety of other applications, such as robotic control and othermachine vision applications, inspection systems and medical imagerysystems, without departing from the spirit and scope of the presentinvention.

The image correlation method and apparatus 10 of the present inventioncompares a test image with a template. Thus, the image correlationapparatus of the present invention can include template generation meansfor creating a template as shown in FIG. 1. The template generationmeans 12, such as an image processor 46 and associated software, cancreate the template in a variety of manners. For example, a referenceimage can be obtained which is digitized to create the template. Moreparticularly, a photograph, such a reconnaissance photograph, can betaken of the area of interest. The photograph can then be digitallyscanned to create a template formed of a number of pixels. Each pixel isrepresentative of a predetermined gray level, typically designated 0 to255, as known to those skilled in the art. As illustrated in FIG. 1, theimage correlation apparatus also includes template memory means 14, suchas a random access memory device, for storing indicia representative ofthe plurality of pixels which form the template.

Alternatively, the template can be created by an infrared sensor, avisible wavelength sensor, synthetic aperture radar, or any of a varietyof other sensors known to those skilled in the art. Thus, the templategeneration means 12 typically includes a sensor for obtaining areference image as well as any associated conversion devices, such as adigital scanner, for converting the reference image to a digitaltemplate.

The image correlation method and apparatus 10 compares the plurality ofpixels which form the template to a test image, also formed of aplurality of pixels representative of predetermined gray levels. Thus,the image correlation apparatus preferably includes image capturingmeans 16 for capturing the test image. As described above in conjunctionwith the template generation means 12, the test image can be captured ina variety of manners, including the digital scanning of reconnaissanceor surveillance photographs, or the creation of a test image by infraredsensors, visible wavelength sensors, synthetic aperture radar or any ofa variety of other sensing techniques known to those skilled in the art.Thus, the image capturing means typically includes a sensor forobtaining a reference image as well as any associated conversiondevices, such as a digital scanner, for converting the reference imageto a digital template. As shown in FIG. 1, the image correlationapparatus also includes image memory means 18, such as a random accessmemory device, for storing indicia representative of the plurality ofthe pixels which form the test image.

Due to the comparison of the test image to the template provided by theimage correlation method and apparatus 10 of the present invention, thetemplate and the test image can be obtained or created by differenttypes of sensors. For example, the template can be created by digitallyscanning a reconnaissance photograph, while the test image is capturedwith an infrared sensor, a visible wavelength sensor or syntheticaperture radar.

According to the image correlation method and apparatus 10 of thepresent invention, the template is initially partitioned into aplurality of labels, each of which is formed by at least one pixel, asshown in block 50 of FIG. 2. The template can be partitioned into labelsin a variety of manners without departing from the spirit and scope ofthe present invention. However, according to one advantageousembodiment, the image correlation apparatus and, more particularly, thetemplate generation means 12, includes template processing means 20having partitioning means 22 for partitioning a template into a numberof labels.

According to one advantageous embodiment, the partitioning means 22partitions the template into labels based upon the relative gray levelsof the template pixels. Thus, the partitioning means preferablydetermines the number of pixels of the template which are representativeof each of the predetermined gray levels. As shown in FIG. 3, thepartitioning means effectively creates a histogram which depicts thenumber of pixels of the template which are representative of each of thepredetermined gray levels. As shown in block 80 of the flow chart ofFIG. 5 which illustrates the label partitioning process in more detail,the partitioning means can also include means for dividing thepredetermined gray levels into a number of bins. As illustrated in FIG.4, each bin, designated 0-11, includes a contiguous range of graylevels. For example, the means for dividing the gray levels into binscan initially divide the gray levels according to a predeterminedallocation technique, such as allocating an equal number of gray levelsto each bin or allocating predefined ranges of gray levels to therespective bins.

As shown in block 82 of FIG. 5, the partitioning means 22 can alsoinclude means for determining, for each of the bins, an actual count ofthe number of pixels which are representative of gray levels within therange of gray levels included within the respective bin. Thepartitioning means can further include means for assigning apredetermined minimum count to each bin which has an actual count whichis less than the predetermined minimum count. Thus, the partitioningmeans compares the actual count of each bin to the predetermined minimumcount and replaces the actual count with the predetermined minimum countif the actual count is less than the predetermined minimum count asshown in blocks 84 and 86 of FIG. 5.

The predetermined minimum count can be set to any desired value.However, the partitioning means 18 of one embodiment also includes meansfor determining the maximum number of pixels of the template which arerepresentative of a single predetermined gray level. This maximum numberof pixels is graphically represented as the peak of the histogram ofFIGS. 3 and 4. According to this embodiment, the partitioning means canalso include means for computing the predetermined minimum count to beequal to a predetermined percentage of the peak, i.e., the maximumnumber of pixels which are representative of a single predetermined graylevel. In one advantageous embodiment, the predetermined percentage isbetween about 15% and about 25% and, more preferably, is about 20% ofthe peak value.

The partitioning means 22 of this embodiment also includes means forreassigning the plurality of predetermined gray levels to the bins suchthat the greater of the predetermined minimum count or the actual countfor each respective bin is within a predetermined range as shown inblocks 88 and 90 of FIG. 5. For purposes of illustration, the boundariesof the bins depicted in FIG. 4 can be moved inwardly or outwardly todecrease or increase the number of gray level and, therefore, the numberof pixels within a bin, respectively. According to one advantageousembodiment, the gray levels are preferably reassigned to the bins suchthat greater of the predetermined minimum count or the actual count foreach bin is within 5% of the greater of the predetermined minimum countor the actual count for each of the other bins. More preferably, thegray levels are preferably reassigned to the bins such that the greaterof the predetermined minimum count or the actual count for each bin isequal.

This process of counting the number of pixels in each bin, assigning thepredetermined minimum count to bins having an actual count less than thepredetermined minimum count, and reassigning the gray levels cancontinue repeatedly until the greater of the predetermined minimum countor the actual count for each bin is within a predetermined range.Following the reassignment of the gray levels as described above, thepixels which are representative of gray levels within a respective bin,such as those designated 0-11 in FIG. 4, form a label. By way ofexample, FIG. 6 illustrates a template T which has been divided intothree labels, namely, label 1, label 2, and label 3.

Alternatively, a template can be divided into labels based upon thespatial or relative positional relationship of the pixels without strictregard for the respective gray levels of the pixels. In this fashion,the roof of a building could be designated by a first label, while ariver or street is designated by a second label, for example.

As shown in FIG. 1 and in block 52 of FIG. 2, the template processingmeans 20 also preferably includes means 24 for determining the totalnumber of pixels N_(T) which form the template. Thus, for a rectangulartemplate formed of 512×512 pixels, the total number of pixels N_(T) willbe 262,144. In addition, the template processing means preferablyincludes means 26 for determining the number of pixels N_(i) which formeach of the plurality of labels of the template as shown in block 54 ofFIG. 2. For example, the N_(i) determining means would separatelydetermine the number of pixels N₁, N₂ and N₃ which form label 1, label2, and label 3 of the template, respectively.

As shown in FIG. 1 and in block 68 of FIG. 2, the image correlationapparatus 10 also includes comparison means 28, responsive to thetemplate processing means 20, for comparing the test image to thetemplate such that at least some of the pixels of the test imagecorrespond to respective pixels of the template. As shown in FIG. 7, thecomparison means essentially overlays the template T onto the test imageTI and compares the pixels of the template and the test image on apixel-by-pixel basis.

As illustrated in FIG. 1 and in block 70 of FIG. 2, the comparison means28 includes means 30 for determining, for each of the predetermined graylevels j, the number of pixels of the test image N_(j),i representativeof a predetermined gray level j which correspond to a predeterminedlabel i of the template. Thus, for the exemplary comparison illustratedby FIG. 7, the N_(j),i determining means would separately determine thenumber of pixels of the test image which correspond to or fall underlabels 1, 2 and 3 which have each of the predetermined gray levels,typically designated 0-255. More specifically, for the pixels of thetest image which correspond or fall under label 1 of the exemplarytemplate illustrated in FIG. 7, the N_(j),i determining means wouldseparately determine the number of pixels representative of gray level0, gray level 1, gray level 2, . . . gray level 255, which correspond toor fall under label 1 in order to determine N₀,1, N₁,1, N₂,1, . . .N₂₅₅,1, respectively. The N_(j),i determining means would thenseparately determine the number of pixels of the test image whichcorrespond to or fall under label 2 of the template which have each ofthe predetermined gray levels. The N_(j),i determining means would thuscontinue to determine, for each of the predetermined gray levels, thenumber of pixels which correspond to or fall under each of the labels inturn.

As also shown in FIG. 1 and in block 72 of FIG. 2, the comparison means28 also preferably includes means 32 for determining, for each of theplurality of predetermined gray levels j, the number of pixels of thetest image N_(j) representative of the predetermined gray level j whichcorrespond to the template. Thus, in the above example, the N_(j)determining means would separately determine the number of pixels of thetest image representative of gray level 0, gray level 1, gray level 2, .. . gray level 255 which correspond to the template in order todetermine N₀, N₁, N₂, . . . N₂₅₅, respectively.

The image correlation apparatus 10 of the present invention alsoincludes correlation means 34, responsive to the template processingmeans 20, and the comparison means 28, for determining the correlation Xbetween the test image and the template as shown in FIG. 1 and in block74 of FIG. 2. In particular, the correlation means determines thecorrelation X according to the following equation: ##EQU3##

As used in equation (1) above, i_(min) and i_(max) are the minimum andmaximum labels designations, respectively. For the exemplary templateillustrated in FIG. 6 which has three labels designated label 1, label 2which has and label 3, i_(min) would be 1 and i_(max) would be 3.Similarly, j_(min) and j_(max) are the minimum and maximum gray levels,respectively. Typically, the gray levels are designated 0-255 such thatj_(min) is 0 and j_(max) is 255.

The correlation X determined by the correlation means 34 of the imagecorrelation method and apparatus 10 of the present invention effectivelymeasures the correlation or match between the template and the portionof the test image to which the template is compared or overlayed. Thus,greater values of correlation X indicate a greater correlation or matchbetween the test image and the template and, consequently, a higherlikelihood that the reference image represented by the template has beenlocated within the test image.

In order to efficiently and rapidly compute the correlation X betweenthe test image and the template, the image correlation method andapparatus of the present invention and, more particularly, thecomparison means 28 can include means for determining ln N_(j),i and lnN_(j) prior to determining the correlation X. In particular, ln N_(j),iand ln N_(j) can be determined during the process of determining N_(j),iand N_(j). Consequently, the values of ln N_(j) and ln N_(j) for each ofthe labels and for each of the predetermined gray levels j can becomputed and stored, such as in a lookup table within a random accessmemory device, prior to determining the correlation X between the testimage and the template. Thus, these values need not be recomputed duringthe determination of the correlation X, but can, instead, be recalled orlooked up in the appropriate lookup table during the computation of thecorrelation X between the test image and the template.

Due to the comparison of the test image to the template on apixel-by-pixel basis in which each corresponding pixel of the templateand the test image are compared, the image correlation method andapparatus 10 of the present invention is believed to be significantlyless sensitive to noise than conventional image correlation methodswhich employ, for example, edge or boundary detection techniques. Inaddition, due, at least in part, to the comparison and correlation ofthe test image and the template provided by the image correlation methodand apparatus of the present invention, the template and the test imagecan be reliably correlated even though they were obtained or captured bydifferent types of sensors, such as photographic sensors, visiblewavelength sensors, infrared sensors or synthetic aperture radar. Inaddition, the image correlation method and apparatus of the presentinvention is relatively versatile with few, if any, limitations on thetype or size of the structures which can be represented within thetemplate and the test image.

According to one preferred embodiment, the image correlation apparatus10 also includes offset means 36 for creating a plurality of relativeoffsets between the template and the test image as shown in FIG. 1 andin blocks 54 and 56 of FIG. 2. Thus, the template can be compared to thetest image at each of the relative offsets. Moreover, the N_(j),idetermining means 30 can determine, at each relative offset, the numberof pixels of the test image representative of a gray level j whichcorrespond to a predetermined label i of the template. Likewise, theN_(j) determining means 32 can determine, at each relative offset, thenumber of pixels of the test image representative of predetermined graylevel j which correspond to the template. Consequently, the correlationmeans 34 can determine the correlation X between the test image and thetemplate at each relative offset. As will be apparent to those skilledin the art, the test image and the template may correlate or match muchbetter at one or more relative offsets than others. The imagecorrelation apparatus of the present invention can therefore alsoinclude means 38 for determining the relative offset between the testimage and the template which provides the greatest correlationtherebetween.

According to one embodiment, the pixels which form the test image andthe template are assigned respective addresses. For example, the pixelsare typically assigned an address having X and Y components whichrepresent the relative offset of the respective pixels from a referencepoint, such as one corner of the test image or template, as shown inFIG. 7. According to this embodiment, the image correlation apparatus 10can also include an address generator 40. The address generatortypically includes the offset means 36 as shown in FIG. 1 so as tocreate the relative offset between the template and the test image basedupon the respective addresses of the pixels which form the template andthe test image.

The address generator 40 can create relative offsets between thetemplate and the test image such that the template is compared to anynumber of different regions of the test image without departing from thespirit and scope of the present invention. It is generally preferred tocompare the template with a large number of regions of the test imagesuch that the greatest correlation between the template and test imagecan be more precisely determined. However, the length of time requiredfor this comparison process generally increases as the template iscompared to more regions of the test image.

For purposes of illustration, however, the address generator 40 canincrementally increase the relative offset in the X- direction from 0 tothe maximum width of the test image for each offset in the Y-directionfrom 0 to the maximum height such that the template is compared withevery possible region of the test image. Alternatively, the addressgenerator can create relative offsets such that the template is comparedto regions of the test image which have a greater difference orseparation therebetween in order to expedite the comparison process.Although the comparison of the template to the test image at each of therelative offsets can be performed sequentially, these comparisons arepreferably performed concurrently in order to further expedite the imagecorrelation process, as described hereinafter. In addition, the testimage could be moved relative to a stationary template in order tocreate the relative offset therebetween without departing from thespirit and scope of the present invention.

As shown in blocks 58 and 60 of FIG. 2, the image correlation apparatus10 of the present invention and, more particularly, the addressgenerator 40 can also include rotation means 42 for creating at leastone relative rotational offset α between the template and the testimage. As shown in FIG. 8, for example, a relative rotational offset iscreated between the template and the test image. While the template isrotated relative to the test image in FIG. 8, the template can, instead,be rotated relative to the test image without departing from the spiritand scope of the present invention.

As described above in conjunction with the offset means 36, the testimage can then be compared to the template at each relative rotationaloffset. In this regard, the N_(j),i determining means 30 can determine,at each relative rotational offset, the number of pixels of the testimage N_(j) representative of a predetermined gray level j whichcorrespond to a predetermined label i of the template. In addition, theN_(j) determining means 32 can determine, at each relative rotationaloffset, the number of pixels of the test image N_(j) representative of apredetermined gray level j which correspond to the template.Consequently, the correlation means 34 can determine the correlation Xbetween the test image and the template at each relative rotationaloffset.

The image correlation apparatus 10 of this embodiment can also includemeans 38 for determining the relative rotational offset between the testimage and the template which provides the greatest correlationtherebetween. Thus, the template and the test image can be compared at anumber of relative offsets in order to determine the relative positionsof the template and the test image which provides the greatestcorrelation or the best match therebetween.

In order to provide even greater versatility to the image correlationmethod of the present invention, the image correlation apparatus 10 and,more particularly, the address generator 40, can also include scalingmeans 44 for scaling the template relative to the test image accordingto at least one predetermined scale as shown in blocks 62 and 64 of FIG.2. As illustrated in FIG. 9, the template can be increased in scale orenlarged relative to the test image. However, the template can, instead,be reduced in scale relative to the test image without departing fromthe spirit and scope of the present invention. Alternatively, thescaling means can scale the test image relative to the template, i.e.,can enlarge or reduce the size of the test image relative to thetemplate, instead of enlarging or reducing the size of the template,without departing from the spirit and scope of the present invention.

The image correlation method and apparatus 10 of this embodiment alsopreferably compares the test image to the template at each predeterminedscale. In particular, the N_(j),i determining means 30 can determine, ateach predetermined scale, the number of pixels of the test image N_(j),irepresentative of a predetermined gray level j with correspond to apredetermined level i of the template. Likewise, the N_(j) determiningmeans 32 can include means for determining, at each predetermined scale,the number of pixels of the test image N_(j) representative of apredetermined gray level j with correspond to a predetermined level i ofthe template. Consequently, the correlation means 34 can determine thecorrelation X between the test image and the template at eachpredetermined scale. Furthermore, the image correlation apparatus caninclude means 38 for determining the predetermined scale which providesthe greatest correlation or match between the template and the testimage. Thus, the image correlation method and apparatus of the presentinvention can effectively correlate or match test images and templateswhich are initially scaled differently, thereby further increasing theversatility and functionality of the image correlation method andapparatus of the present invention.

While, the generation of relative offsets, relative rotational offsetsand scaling factors by the address generator 40 and, more particularly,by the offset means 36, the rotation means 42 and the scaling means 44,respectively, have been separately discussed hereinabove, the relativepositions and sizes of the test image and the template can be adjustedby a combination of two or all three of the foregoing offsets orscaling. Thus, the template can be both rotated and scaled by apredetermined number of angles and scales relative to the test imagewithout departing from the spirit and scope of the present invention.Thus, the image correlation method and apparatus 10 can more preciselydetermine the relative positions and sizes of the test image and thetemplate which produce the greatest correlation or match therebetween.

According to one advantageous embodiment illustrated in FIG. 10, thecomparison means 34 can include a plurality of processing elements 92responsive to the template memory means 14 and the image memory means18. As shown, each processing element preferably includes a fieldprogrammable gate array 94 for comparing the template to the test imageand one or more memory devices 96, such as random access memory devices.However, the processing elements can, instead, include a combination ofsoftware and hardware, such as a microprocessor, a gate array, anapplication specific integrated circuit or a custom integrated circuitwithout departing from the spirit and scope of the present invention.More particularly, each processing element preferably includes theN_(j),i determining means 30 and the N_(j) determining means 32 asdescribed above. In addition, each processing element can include thecorrelation means 34 for determining the correlation X between the testimage and the template according to equation (1).

As illustrated in FIG. 10, each processing element 92 can include thetest image memory means 18 and can be operably connected to the templatememory means 14 such that the pixels representative of the test imageand the template can be compared by each of the processing elements.According to the present invention, the processing elements arepreferably disposed in parallel such that the comparison of the testimage and the template can be performed concurrently by the processingelements.

In addition, the image correlation apparatus 10 illustrated by FIG. 10includes an address generator 40, including the offset means 36,rotation means 42 and scaling means 44, for creating a plurality ofrelative offsets between the template and the test image. Thus, thepixels which form the template can be provided to the processingelements 92 along with the respective offsets of the template to thetest image. In particular, the address generator 40 can supply pixeladdresses to the respective processing elements which specify or definethe relative positions of the template pixels to the test image pixelsand consequently, determines the relative offset therebetween. Inaddition, the address generator can provide a different set of pixeladdresses to each of the processing elements such that the processingelements compare versions of the template which have been offset bydifferent amounts to the test image. Consequently, processing element 0can compare the template to the test image with no relative offsettherebetween, processing element 1 can compare the template to the testimage with a first relative offset therebetween, processing element 2can compare the template and the test image with a second relativeoffset therebetween, and so on.

As described above, the address generator 40 can also include therelative rotation means 42 and scaling means 44 for providing a relativerotational offset between the template and the test image and forscaling the template relative to the test image. In particular, theaddress generator can supply pixel addresses to the respectiveprocessing elements 92 which specify or define the relative positions ofthe template pixels to the test image pixels and, consequently,determines the relative rotation or scaling therebetween. In addition,the address generator can provide a different set of pixel addresses toeach of the processing elements such that the processing elementscompare versions of the template which have been rotated or scaled bydifferent amounts to the test image.

For example, processing element 0 can compare the test image to thetemplate with no relative rotational offset therebetween, processingelement 1 can compare the test image to the template with a firstrelative rotational offset therebetween, processing element 2 cancompare the test image to the template with a second relative rotationaloffset therebetween, and so on. Likewise, processing element 0 cancompare the template to the test image with no relative scaling of thetemplate to the test image, processing element 1 can compare thetemplate to the test image with a first relative scaling of the templateto the test image, processing element 2 can compare the template to thetest image with a second relative scaling of the template to the testimage, and so on. As described above, by concurrently performing each ofthese comparisons of the test image to the template, the imagecorrelation method and apparatus 10 of the present invention efficientlyand rapidly correlates or matches the test image and the template.

Each processing element 92 also preferably includes an additional memorydevice 96, such as a random access memory device, which serves, amongother functions, as a scratchpad during the determination of N_(j),i andN_(j) as well as the determination of the correlation X between the testimage and the template. Following the comparison of the test image andthe template, the correlation X can be stored by the processing element,such as within the additional memory device of the respective processingelement.

Alternatively, the correlation X can be transmitted to the systemprocessor 98 for storage and subsequent processing. In particular, thesystem processor can include the means 38 for determining the greatestcorrelations between the test image and the template. Thus, the systemprocessor can compare the respective correlations X determined by eachof the processing elements at each of the relative offsets, relativerotational offsets and/or relative scales to determine the relativepositions of the test image and the template which provides the greatestcorrelation therebetween. As schematically depicted in FIG. 10, thesystem processor is preferably operably connected to one or more sensorsvia a sensor image interface 97 and to a computer network or system viaa system interface 99 in order to provide operator interaction with theimage correlation apparatus as desired.

The test image and the reference image from which the template isconstructed are oftentimes obtained from different viewpoints, such asfrom different angles and/or from different directions. For example, thetemplate and the test image can be obtained from first and secondviewpoints, respectively. For the missile guidance applicationsillustrated in FIG. 11, a navigation system 10 can monitor and determinethe first and second viewpoint at which the template and the test imagewere obtained.

In order to at least partially compensate for the differences betweenthe first and second viewpoints, the image processor 66 which forms thetemplate generator means 12 can geometrically warp the template. Thisgeometric warping is typically performed prior to storing indiciarepresentative of the pixels which form the template in the templatememory means 14 as shown in blocks 102 and 108 of FIG. 11, respectively.As known to those skilled in the art, the geometric warping of an imageis typically performed by means of transformation viewpoint equationswhich are implemented by the image processor. For a more detaileddescription of geometric warping see a book entitled Computer Graphics:Principles and Practice by Foley et al., published by Addison-WesleyPublishing Company, Inc., Reading, Massachusetts (1990).

Likewise, the image processor 46 can texture map the template to furthercompensate for differences between the first and second viewpoints asshown in block 104 of FIG. 11. The template is also preferably texturemapped by the image processor prior to be being stored within thetemplate memory means 14. By texture mapping the template, localizeddetails within the template or within the reference image from which thetemplate is constructed are morphed onto a three-dimensional shape. Amore detailed description of texture mapping is also provided by thebook entitled Computer Graphics: Principles and Practice. Since thegeometric warping and texture mapping compensate for differences betweenthe first and second viewpoints, it is generally unnecessary togeometrically warp and/or texture map the template prior to storingindicia representative of the pixels which form the template in thetemplate memory means in instances in which the first and secondviewpoints are closely related.

The image processor 40 can also remove at least some of the backgroundshading from the template in order to further improve the imagecorrelation process as shown in block 106 of FIG. 11. The removal of atleast some of the background shading is also preferably performed priorto storing indicia representative of the pixels which form the templatein the template memory means 14. The image processor can remove at leastsome of the background shading from the template according to any methodknown to those skilled in the art. However, according to oneadvantageous embodiment, the image processor removes at least some ofthe background shading by determining the average gray level of thepixels in a predetermined region about each individual pixel. Thisaverage gray level value is then subtracted from the gray level of theindividual pixel. Once this process has been repeated for each pixel ofthe template, a predetermined value, can be added to the gray level ofeach pixel in order to create a template in which at least some of thebackground shading has been removed.

The image correlation method and apparatus 10 image of the presentinvention also provides for adaptive deformation between the templateand the test image to further increase the versatility and effectivenessof the present invention. As illustrated schematically in block 110 ofFIG. 11, the image processor 40 can divide the template into a number ofpieces, such as nine equal-sized pieces. Each of the template pieces canthen be compared to a predetermined region of the test image in order todetermine the correlation of the template pieces to the test image,while allowing at least some deformation between the pieces of thetemplate, as shown in block 112 of FIG. 11.

According to this embodiment, the N_(j),i determining means 30 caninclude means for separately determining, for each piece of thetemplate, the number of pixels of the test image N_(j),i representativeof a predetermined gray level j which correspond to a predeterminedlabel i of the respective piece of the template. Likewise, the N_(j)determining means 32 of this embodiment can include means for separatelydetermining, for each piece of the template, the number of pixels of thetest image N_(j) representative of a predetermined gray level j whichcorrespond to the respective piece of the template. Based upon thevalues of N_(j),i and N_(j) which have been determined separately foreach piece of the entire template, the correlation means 34 canseparately determine the correlation of each piece the template to thetest image according to equation (1).

Each piece of the template is preferably compared to a portion of thetest image within the predetermined range of the test image which isassigned to the respective piece of the template. As describedhereinafter, the predetermined regions of the test image are preferablyselected such that the pieces of the template can partially overlap orcan be spaced apart. Thus, the template as a whole can be compared andcorrelated with the test image, while allowing the individual pieces ofthe template to move slightly relative to one another, thereby furthercompensating for deformations, if any, between the template and the testimage.

For purposes of illustration, three pieces of the template illustratedin FIG. 6 are shown in FIG. 12. As shown, the template is preferablydivided into pieces such that those portions of the test image along theborders or boundaries of a first piece of the template are also includedalong the borders of a second piece of the template, adjacent the firstpiece. Thus, at least some portions of the test image will be includedalong the borders or boundaries of two or more pieces of the template toinsure that all portions of the test image are compared to the template.However, the template can be evenly divided into pieces which do notcontain common portions of the test image along their borders withoutdeparting from the spirit and scope of the present invention.

FIG. 12 also illustrates the predetermined range of the test imagewithin which each of the three exemplary pieces of the template can becompared by the rectangular boxes surrounding the template pieces. Forillustrative purposes, the rectangular boxes which define the respectivepredetermined ranges of the three pieces of the template have beendesignated with a number of x's, circles and triangles, respectively. Itshould be apparent to one skilled in the art, however, that therectangular boxes and the associated designations form no portion of thetemplate or the test image, but are merely shown in FIG. 12 toillustrate the size of the respective predetermined ranges relative tothe pieces of the template and to graphically depict the overlap betweenthe predetermined ranges.

As described above, these predetermined regions at least partiallyoverlap such that the respective template pieces can also at leastpartially overlap. As shown in FIG. 12, the template pieces arepreferably smaller in size than the predetermined region in which theycan be positioned. Consequently, the pieces of the template can becompared to portions of the test image within the predetermined rangewithout overlapping an adjacent piece of the template, therebypermitting a slight gap between the adjacent pieces of the template.

According to this embodiment of the present invention, the imagecorrelation apparatus 10 also preferably includes offset means 36 forcreating a number of relative offsets between the pieces of the templateand the test image. Thus, by appropriately selecting the relativeoffsets between the pieces of the template and the test image, thepieces of the template can be compared to several different portions ofthe test image within the predetermined range. Consequently, thecomparison means 28 can separately determine for each piece of thetemplate the values of N_(j),i and N_(j) at each relative offset and theresulting correlation X between each piece of the template and the testimage at each relative offset. The comparison means can then separatelydetermine for each piece of the template the relative offset whichprovides the greatest correlation X between the respective piece of thetemplate and the test image. Thereafter, the comparison means can sumthe greatest correlation X determined for each piece of the template todetermine the resulting correlation of the template, which is formed ofa number of pieces, to the test image.

The offset means 36 can also create a number of relative offsets betweenthe template, as a whole, and the test image. In particular, the imagecorrelation apparatus 10 of the present invention can determine thecorrelation in the manner described above between a template, which isformed of a number of pieces, and the test image which is positioned ina first relative position thereto. Thereafter, the image correlationapparatus can determine the correlation in the manner described abovebetween the template and the test image which is positioned in a secondrelative position thereto by the offset means. Accordingly, the imagecorrelation apparatus of the present invention can determine thecorrelation between the template, which is formed of a number of pieces,and the test image at a number of predetermined relative positions suchthat the relative position between the template and the test image whichprovides the greatest correlation therebetween can be determined asdescribed below.

Likewise, the image correlation method and apparatus 10 of thisembodiment can also include rotation means 42 and scaling means 44 asdescribed above such that the correlation X can be determined betweenthe test image and the template at each relative rotational offset andat each relative scaling. For each relative rotational offset and foreach relative scaling, the image correlation apparatus preferablydetermines the relative position of each piece of the template whichprovides the greatest correlation between the respective piece of thetemplate and the test image in the manner described above. The overallcorrelation of the template to the test image can then be determined ateach relative rotational offset and each relative scaling by summing thegreatest correlation of each respective piece of the template. Thus, theimage correlation method and apparatus of the present invention candetermine the relative offset, relative rotational offset and/orrelative scale between the test image and the template, which is formedof a number of pieces, which provides the greatest correlationtherebetween.

In this regard, the image correlation method and apparatus 10 of thisembodiment of the present invention can also include means 38, such asthe system processor 40, for determining the relative offset between thetest image and the template which provides the greatest correlationtherebetween. If this greatest correlation between the test image andthe template is greater than a predetermined minimum correlation, thetest image and the template are considered to have been matched and apreselected object designated within the template can be located andidentified within the test image as described hereinbelow.

In one advantageous embodiment illustrated in FIG. 10 in which thetemplate is positioned at a predetermined relative offset, apredetermined relative rotational offset and a predetermined relativescale to the test image, the plurality of processing elements 92 canconcurrently compare the same piece of the template to the test image ata variety of relative offsets within the predetermined range of the testimage assigned to the respective piece in order to determine thegreatest correlation therebetween. Subsequently, the plurality ofprocessing elements can concurrently compare a second piece of thetemplate to the test image at a variety of respective relative offsetswithin the predetermined range of the test image assigned to the secondpiece in order to determine the greatest correlation therebetween. Thiscomparison process can continue until each of the pieces of the templatehas been compared to their respective predetermined range of the testimage. The respective greatest correlations of each individual piece ofthe template can then be summed to determine the overall correlationbetween the template and the test image at the predetermined relativeoffset, the predetermined relative rotational offset and thepredetermined relative scale. Thereafter, the comparison process can berepeated with the template positioned at other predetermined offsets,other predetermined rotational offsets and/or other predetermined scalesrelative to the test image in order to determine the overall correlationbetween the template and the test image at each of these other offsetsand/or scales.

In order to further increase the probability that the template iscorrectly correlated or matched with a portion of the test image, theimage correlation method and apparatus 10 of the present invention cancorrelate a number of temporally distinct test images with the template.The image correlation method and apparatus can then weight the resultsto more accurately match the template with a portion of the test image.Thus, the test image capturing means 16 of this embodiment preferablyincludes means for capturing a predetermined number of temporallydistinct test images, such as ten test images captured in 1 secondincrements. According to this embodiment, the comparison means 28, suchas the plurality of processing elements 92, includes means forseparately comparing each of the temporally distinct test images to thetemplate as shown schematically in block 114 of FIG. 11. Consequently,the N_(j),i determining means can include means for separatelydetermining the number of pixels of the temporally distinct test imagesN_(j),i representative of a predetermined gray level j which correspondto a predetermined label i of the template. Likewise, the N_(j)determining means can include means for separately determining thenumber of pixels of the temporally distinct test images N_(j)representative of a predetermined gray level j which correspond to thetemplate. Thus, the correlation means 34 can separately determine thecorrelation X between each of the temporally distinct test images andthe template.

Additionally, the means 38 for determining the relative offset betweenthe test image and the template which provides the greatest correlationX can include means for determining, for each relative offset, a sum ofthe correlations X determined for each of the temporally distinct testimages which have been weighted according to a predetermined time-basedformula. The correlations determined for each of the temporally distincttest images at a respective offset can be weighted according to avariety of formulas without departing from the spirit and scope of thepresent invention. However, the time-based formula of one advantageousembodiment weights the correlations determined for each of thetemporally distinct test images at a respective offset such thatcorrelations X determined for the more recent test images are assigned ahigher or great weight than the older or more distant test images.

For example, for a number of temporally distinct test images designatedTI₁, TI₂, . . . TI₁₀ in which TI₁ is the oldest test image and TI₁₀ isthe newest test image, the respective correlations X determined for eachof these temporally distinct test images at a respective offset can beweighted such that the correlations X determined for the more recenttest images are assigned a higher or greater weight than the older ormore distant test images. For example, for a number of temporallydistinct test images designed TI₁, TI₂, . . . TI₁₀ in which TI₁ is theoldest test image and TI₁₀ is the newest test image, the respectivecorrelations X determined for each of these temporally distinct testimages can be weighted by being multiplied by 0.1, 0.2, 0.3, . . . 1.0,respectively. Thus, the correlations X associated with the more recenttest images are weighted more heavily and contribute more significantlyto the resulting correlation X of the series of test images to thetemplate. As described above, the relative offset between the test imageand the template which provides the greatest sum of weightedcorrelations X over time can then be determined in order to moreprecisely match or correlate the template to a portion to the testimage.

Although the image correlation method and apparatus 10 can be employedin a variety of applications without departing from the spirit and scopeof the present invention, the image correlation method and apparatus 10of the present invention can form a portion of an object recognitionmethod and apparatus which is adapted to recognize or identify apreselected object or target within a test image as shown in FIG. 11.Consequently, the object recognition method and apparatus can includeobject designation means 48, responsive to the template generation means12, for designating the selected object or target within the template.Following comparison of the template to the test image and thedetermination of the relative position of the template to the test imagewhich provides the greatest correlation X therebetween, the selectedobject designated within the template can be identified or locatedwithin the test image. In particular, the object designated in thetemplate can be identified or located within the portion of the testimage which provides the greatest correlation or match with thetemplate. In particular, the object recognition method and apparatus ofthe present invention can include object recognition means 49 forselecting an object within the test image at the relative offset whichprovides the greatest correlation or match with the template whichcorresponds to the designated object in the template. Consequently, apreviously selected object, such as a target designated in areconnaissance photograph or a preselected part of a complicatedmechanical assembly, can be thereafter reliably identified or locatedwithin a test image.

Therefore, the image correlation method and apparatus 10 of the presentinvention can rapidly compare a test image to a template to determinethe correlation X therebetween with little, if any, modeling ormensuration required. In addition, the image correlation method andapparatus of the present invention can reliably correlate test imagesand templates which were obtained under different lighting or otherenvironmental conditions or which were obtained from cross-spectralsources. Further, the image correlation method and apparatus of thepresent invention can reliably correlate a test image with a templateregardless of the type, shape, location or material of the objectdesignated within the template. Consequently, the image correlationmethod and apparatus of the present invention provides a versatile,reliable and rapid comparison of a test image with a template such thatobjects which were previously designated within the template can bequickly identified in the test image.

In the drawings and the specification, there has been set forthpreferred embodiments of the invention and, although specific terms areemployed, the terms are used in a generic and descriptive sense only andnot for purpose of limitation, the scope of the invention being setforth in the following claims.

What is claimed is:
 1. A method for correlating a test image with a template to identify a specific object wherein both the test image and the template are comprised of a plurality of pixels representative of respective predetermined gray levels, and wherein the template is partitioned into a plurality of labels comprised of at least one pixel, the method comprising the steps of:dividing the template into a plurality of pieces; comparing each piece of the template to a predetermined region of the test image such that at least some of the pixels of each piece of the template correspond to respective pixels of the test image and such that at least some deformation between the pieces of the template is permitted; separately determining, for each piece of the template and for each of a plurality of predetermined gray levels j, the number of pixels of the test image N_(j),i representative of a predetermined gray level j which correspond to a predetermined label i of the respective piece of the template; separately determining, for each piece of the template and for each of a plurality of predetermined gray levels j, the number of pixels of the test image N_(j) representative of a predetermined gray level j which correspond to the respective piece of the template to identify said object; and separately determining the correlation X between each piece of the test image and the template according to: ##EQU4## wherein N_(T) is the total number of pixels which comprise the respective piece of the template, N_(i) is the number of pixels which form each of the plurality of labels i of the respective piece of the template, i_(min) and i_(max) are the minimum and maximum label designations, respectively, and j_(min) and j_(max) are the minimum and maximum gray levels, respectively.
 2. A method according to claim 1 further comprising the step of determining an overall correlation between the test image and the template, said overall correlation determining step comprising the step of summing the respective correlation determined for each piece of the template.
 3. A method according to claim 1 further comprising the step of creating a plurality of relative offsets between the pieces of the template and the test image, wherein said comparing step comprises the step of comparing the pieces of the template to the test image at each relative offset, wherein said N_(j),i determining step comprises the step of determining, for each piece of the template and at each relative offset, the number of pixels of the test image N_(j),i representative of a predetermined gray level j which correspond to a predetermined label i of the respective piece of the template, wherein said N_(j) determining step comprises the step of determining, for each piece of the template and at each relative offset, the number of pixels of the test image N_(j) representative of a predetermined gray level j which correspond to the respective piece of the template, and wherein said correlation determining step comprises the step of determining the correlation X between the test image and each piece of the template at each relative offset.
 4. A method according to claim 3 further comprising the step of separately determining the relative offset between the test image and each piece of the template which provides the greatest correlation therebetween.
 5. A method according to claim 4 further comprising the step of determining an overall correlation between the test image and the template, said overall correlation determining step comprising the step of summing the respective greatest correlation determined for each piece of the template.
 6. A method according to claim 3 wherein said relative offset creating step comprises creating respective relative offsets for each piece of the template which permit adjacent pieces of the template to partially overlap.
 7. A method according to claim 3 wherein said relative offset creating step comprises creating respective relative offsets for each piece of the template which permit adjacent pieces of the template to be spaced apart by a predetermined gap.
 8. An apparatus for correlating a test image and a template to identify a specific object wherein both the test image and the template are comprised of a plurality of pixels representative of respective predetermined gray levels, and wherein the template is partitioned into a plurality of labels comprised of at least one pixel, the apparatus comprising:an image processor for dividing the template into a plurality of pieces; comparison means, responsive to said image processor, for comparing each piece of the template to a predetermined range of the test image such that at least some of the pixels of each piece of the template correspond to respective pixels of the test image and such that at least some deformation between the pieces of the template is permitted, wherein said comparison means comprises:means for separately determining, for each piece of the template and for each of a plurality of predetermined gray levels j, the number of pixels of the test image N_(j),i representative of a predetermined gray level j which correspond to a predetermined label of the respective piece of the template; means for separately determining, for each piece of the template and for each of a plurality of predetermined gray levels j, the number of pixels of the test image N_(j) representative of a predetermined gray level j which correspond to the respective piece of the template; and correlation means, responsive to said template processing means and said comparison means, for separately determining the correlation X between the test image and each piece of the template to identify said object according to: ##EQU5## wherein N_(T) is the total number of pixels which comprise the respective piece of the template, N_(i) is the number of pixels which form each of the plurality of labels i of the respective piece of the template, i_(min) and i_(max) are the minimum and maximum label designations, respectively, and wherein j_(min) and j_(max) are the minimum and maximum gray levels, respectively.
 9. An apparatus according to claim 8 further comprising the means, responsive to said correlation means, for determining an overall correlation between the test image and the template, said overall correlation determining means comprising means for summing the respective correlation determined for each piece of the template.
 10. An apparatus according to claim 8 further comprising offset means for creating a plurality of relative offsets between the pieces of the template and the test image, wherein said comparison means is responsive to said offset means for comparing the pieces of the template to the test image at each relative offset, wherein said N_(j),i determining means comprises means for determining, for each piece of the template and at each relative offset, the number of pixels of the test image N_(j),i representative of a predetermined gray level j which correspond to a predetermined label i of the respective piece of the template, wherein said N_(j) determining means comprises means for determining, for each piece of the template and at each relative offset, the number of pixels of the test image N_(j) representative of a predetermined gray level j which correspond to the respective piece of the template, and wherein said correlation means comprises means for determining the correlation X between the test image and each piece of the template at each relative offset.
 11. An apparatus according to claim 10 further comprising means, responsive to said correlation means, for separately determining the relative offset between the test image and each piece of the template which provides the greatest correlation therebetween.
 12. An apparatus according to claim 11 further comprising the means, responsive to said greatest correlation determining means, for determining an overall correlation between the test image and the template, said overall correlation determining means comprising means for summing the respective greatest correlation determined for each piece of the template.
 13. An apparatus according to claim 10 wherein said offset means creates respective relative offsets for each piece of the template which permit adjacent pieces of the template to partially overlap.
 14. An apparatus according to claim 10 wherein said offset means creates respective relative offsets for each piece of the template which permit adjacent pieces of the template to be spaced apart by a predetermined gap. 